fix(libc): keep buffered input when a seek target is rejected - #15236

What

fseeko (core/libs/prx/libc/src/FileAndHeap.cpp; fseek and fsetpos call it) handed every seek straight to the host C library. When the target was invalid, both hosts changed the stream before failing:

• Windows: UCRT's _fseeki64 drops the read buffer before its lseek to a negative target fails with EINVAL, so the stream continued at the descriptor's read-ahead offset. After reading 4 of 8 buffered bytes, a rejected fsetpos to -1 left ftello at 8 and the next read at end of file. docs/dev/debt/libc.md recorded this.
• Linux: glibc keeps the buffer, but drops pending ungetc data on the failure path and returns EINVAL instead of EOVERFLOW for a target past OFF_MAX.

FreeBSD 11.4's _fseeko checks the target before touching the stream, and returns -1 with the buffer, pushback and EOF state unchanged when the check fails.

This PR adds a ResolveSeekTarget helper that does the same before either host library sees the call:

• SEEK_CUR: the base is the stream's ftello position. A failure there is returned as is, and a negative base is ESPIPE.
• SEEK_END: the base is the file size from fstat, but only for a regular file whose stream is not in write mode (UCRT's write flag on Windows, __fwriting on glibc). Buffered output would make the logical end larger, and finding it would need a flush. On UCRT, a flush on a read stream discards the buffer, which is the bug.
• Checks: a target past OFF_MAX is EOVERFLOW (84); a negative target is EINVAL (22). A valid target is passed on as SEEK_SET.

The Windows pipe check and the closed-stream and origin checks still come first. The success path is unchanged.

The debt entry now leads with what remains:

• SEEK_END on a stream in write mode, on a non-regular file, or on macOS still goes to the host library;
• a host seek that fails on a valid target is not handled.

Tested

• file_position (core/libs/tests/FilePosition.cpp) gains two cases. Every existing assertion is kept.
  • Rejected seeks: read 4 bytes of an 8-byte file, then try fsetpos to -1, fseeko(-1, SEEK_SET), fseek(-5, SEEK_CUR), fseeko(-9, SEEK_END) and fseeko(INT64_MAX, SEEK_CUR). Each must fail with EINVAL (or EOVERFLOW for the last), leave ftello at 4, and leave EOF and error unchanged. A pushed-back byte must survive a rejected SEEK_CUR and a rejected SEEK_SET, and the next reads must return the original bytes.
  • SEEK_END with pending output: still goes to the host and lands where expected.
  • The test has no platform conditionals.
• Linux (Ubuntu 26.04, GCC 15):
  • with the test on main's FileAndHeap.cpp, file_position fails at line 70 (EINVAL instead of EOVERFLOW);
  • with this PR it passes;
  • the libc stream group (file_position, file_streams, file_stream_status, guest_standard_streams*, guest_stdio_variadic, guest_wide_stream_scan, guest_stream_purge, guest_fopen_errors) passes 12/12.
• Windows 11 (MinGW GCC 15.2.0):
  • with only FileAndHeap.cpp from main (9090972), file_position fails at line 71: ftello is 8, not 4, after the rejected fsetpos;
  • with this PR (4295667) it passes;
  • the same group plus guest_fopen_long_host_path and guest_stream_buffer_host_write passes 13/13, with guest_fopen_errors skipped for lack of symlink rights;
  • no compiler warnings.
• Mutation check: removing the negative-target rejection makes file_position fail at line 86 on Linux, because glibc drops the pushed-back byte. On Windows, removing the SEEK_SET check from the first, Windows-only version of the change failed at the ftello check after the rejected fsetpos.
• python3 tools/check_conventions.py --base origin/main: all conventions checks passed.
• python3 tools/pr_overlap.py: no confirmed overlap. It could not compare 27 PRs that already conflict with main, so their changes were checked by hand: the FileAndHeap.cpp hunks of the eight that edit it, and the libc.md changes of all 27.
  • fix(libc): share multibyte state in fgetwc #10405 (MarcZX10) adds a line after const int nativeError = errno; in fseeko. This PR leaves that line and the ::fseeko line alone.
  • fix(libc): translate native stream write errors #14559 edits the body of fwrite, three lines above the new helper.
  • feat(libc): implement fcloseall with tracked stream lifetimes #11831 adds the same <sys/stat.h> include higher in the include block. If both merge, the include appears twice, which is harmless.
  • fix(libs): share Sony CRT handler and lconv state between libc and libSceAmpr #7081, feat(libc): support archive-backed stdio reads #7447, feat(libc): add std::__sce_v2::_Random_device, _Fiopen and setw #9526, fix(relinker): match AppContent utility import to provider #13486 and fix(libc): translate native stream read errors #14560 edit fopen, freopen, fread and _Fiopen.
  • None of the 27 edits the fseeko and fsetpos debt entry.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: none
• GPUs tested: n/a
• AI-assisted: yes

🤖 Generated with Claude Code